Learning-Based Controller Design with Application to a Chiller Process

In this thesis, we present and study a few approaches for constructing controllers for uncertain systems, using a combination of classical control theory and modern machine learning methods. The thesis can be divided into two subtopics. The first, which is the focus of the first two papers, is dual control. The second, which is the focus of the third and last paper, is multiple-input multiple-output (MIMO) control of a chiller process. In dual control, the goal is to construct controllers for uncertain systems that in expectation minimize some cost over a certain time horizon. To achieve this, the controller must take into account the dual goals of accumulating more information about the process, by applying some probing input, and using the available information for controlling the system. This is referred to as the exploration-exploitation trade-off. Although optimal dual controllers in theory can be computed by solving a functional equation, this is usually intractable in practice, with only some simple special cases as exceptions. Therefore, it is interesting to examine methods for approximating optimal dual control. In the first paper, we take the approach of approximating the value function, which is the solution of the functional equation that can be used to deduce the optimal control, by using artificial neural networks. In the second paper, neural networks are used to represent and estimate hyperstates, which contain information about the conditional probability distributions of the system uncertainties. The optimal dual controller is a function of the hyperstate, and hence it should be useful to have a representation of this quantity when constructing an approximately optimal dual controller. The hyperstate transition model is used in combination with a reinforcement learning algorithm for constructing a dual controller from stochastic simulations of a system model that includes models of the system uncertainties. In the third paper, we suggest a simple reinforcement learning method that can be used to construct a decoupling matrix that allows MIMO control of a chiller process. Compared to the commonly used single-input single-output (SISO) structures, these controllers can decrease the variations in some system signals. This makes it possible to run the system at operating points closer to some constraints, which in turn can enable more energy-efficient operation

Distributed Control of Dynamic Flows in Traffic Networks

In today’s society, traffic congestion is a major problem in several aspects. Apart from the obvious problem that people are losing valuable time due to the resulting delays, it also has negative impact on as well the economy as the local and global environment. With the development of sensors and navigation support, it has now become possible and thus of interest to study optimal routing of vehicles in a traffic network, in order to reduce the congestion-related problems. In this master’s thesis, a distributed algorithm for solution of optimal dynamic traffic flow control problems is derived, implemented and tested. Traffic networks are modelled with the cell transmission model (CTM), and the solution algorithm is based on a generalization of the alternating direction method of multipliers (ADMM). The algorithm is tested for one simple and one more complicated traffic network. The tests include both cases with time-varying external inflow of traffic as well as cases where the flow capacity of a specific road segment is varied with time, in order to simulate temporary traffic incidents. The tests show that if the cost function is chosen as the sum of squares of the traffic volumes at the cells (road segments) of the network, the algorithm converges to the optimal solution if a specific parameter (the penalty parameter, or step length) is chosen sufficiently small. The report starts with a description and examples from the simpler case of static traffic flow optimization. It also contains a summary of the concepts used from optimization theory. After this, the approach for dynamic traffic flow modelling and optimization is described. Finally, a description and derivation of the algorithm is provided, after which the implementation is tested for different cases involving the two different traffic networks

Dual Control of Linear Discrete-Time Systems with Time-Varying Parameters

We describe how the optimal dual controller for a discrete-time linear system can be found by approximately solving the corresponding Bellman equation using a neural network to represent the value function. We illustrate the method on an example with time-varying dynamics, where the new method is shown to give improved performance

Dual Control by Reinforcement Learning Using Deep Hyperstate Transition Models

In dual control, the manipulated variables are used to both regulate the system and identify unknown parameters. The joint probability distribution of the system state and the parameters is known as the hyperstate. The paper proposes a method to perform dual control using a deep reinforcement learning algorithm in combination with a neural network model trained to represent hyperstate transitions. The hyperstate is compactly represented as the parameters of a mixture model that is fitted to Monte Carlo samples of the hyperstate. The representation is used to train a hyperstate transition model, which is used by a standard reinforcement learning algorithm to find a dual control policy. The method is evaluated on a simple nonlinear system, which illustrates a situation where probing is needed, but it can also scale to high-dimensional systems. The method is demonstrated to be able to learn a probing technique that reduces the uncertainty of the hyperstate, resulting in improved control performance

Model-free MIMO control tuning of a chiller process using reinforcement learning

The performance of HVAC equipment, including chillers, is continuing to be pushed to theoretical limits, which impacts the necessity for advanced control logic to operate them efficiently and robustly. At the same time, their architectures are becoming more complex; many systems have multiple compressors, expansion devices, evaporators, circuits, or other elements that challenge control design and resulting performance. In order to maintain respectful controlled speed of response, stability, and robustness, controllers are becoming more complex, including the move from thermostatic control, to proportional integrator (PI), and to multiple-input multiple-output (MIMO) controllers. Model-based control design works well for their synthesis, while having accurate models for numerous product variants is unrealistic, often leading to very conservative designs. To address this, we propose and demonstrate a learning-based control tuner that supports the design of MIMO decoupling PI controllers using online information to adapt controller coefficients from an initial guess during commissioning or operation. The approach is tested on a physics-based model of a water-cooled screw chiller. The method is able to find a controller that performs better than a nominal controller (two single PI controllers) in terms of decreasing deviations from the operating point during disturbances while still following reference changes

On Distributed Optimal Control of Traffic Flows in Transportation Networks

We propose and analyze distributed computation algorithms for finite-horizon optimal control problems in transportation networks. We model traffic flow dynamics by the cell-transmission model and focus on two problems: system-optimum dynamic traffic assignment (where the routing is part of the optimization) and freeway network control (where the routing is exogenous and the optimization is confined to speed limits and ramp-metering controls). While these are non-convex problems, we focus on some recently proposed provably exact convex relaxations and apply Alternating Direction Method of Multipliers techniques. We present fully distributed iterative algorithms and implement them on some transportation network testbeds, testing their convergence speed and accuracy

An extra priming dose of hepatitis A vaccine to adult patients with rheumatoid arthritis and drug induced immunosuppression - A prospective, open-label, multi-center study

Previous studies have indicated that a pre-travel single dose of hepatitis A vaccine is not sufficient as protection against hepatitis A in immunocompromised travelers. We evaluated if an extra dose of hepatitis A vaccine given shortly prior to traveling ensures seroconversion.; Patients with rheumatoid arthritis (n = 69, median age = 55 years) treated with Tumor Necrosis Factor inhibitor(TNFi) and/or Methotrexate (MTX) were immunized with two doses of hepatitis A vaccine, either as double dose or four weeks apart, followed by a booster dose at six months. Furthermore, 48 healthy individuals, median age = 60 years were immunized with two doses, six months apart. Anti-hepatitis A antibodies were measured at 0, 1, 2, 6, 7 and 12 months.; Two months after the initial vaccination, 84% of the RA patients had protective antibodies, compared to 85% of the healthy individuals. There was no significant difference between the two vaccine schedules. At twelve months, 99% of RA patients and 100% of healthy individuals had seroprotective antibodies.; An extra priming dos of hepatitis A vaccine prior to traveling offered an acceptable protection in individuals treated with TNFi and/or MTX. This constitutes an attractive pre-travel solution to this vulnerable group of patients